Search for: All records

Reinforcement Learning from Human Feedback (RLHF) has become the predominant approach for language model (LM) alignment. At its core, RLHF uses a margin-based loss for preference optimization, specifying ideal LM behavior only by the difference between preferred and dispreferred responses. In this paper, we identify a common pitfall of margin-based methods -- the under-specification of ideal LM behavior on preferred and dispreferred responses individually, which leads to two unintended consequences as the margin increases: (1) The probability of dispreferred (e.g., unsafe) responses may increase, resulting in potential safety alignment failures. (2) The probability of preferred responses may decrease, even when those responses are ideal. We demystify the reasons behind these problematic behaviors: margin-based losses couple the change in the preferred probability to the gradient of the dispreferred one, and vice versa, often preventing the preferred probability from increasing while the dispreferred one decreases, and thus causing a synchronized increase or decrease in both probabilities. We term this effect, inherent in margin-based objectives, gradient entanglement. Formally, we derive conditions for general margin-based alignment objectives under which gradient entanglement becomes concerning: the inner product of the gradients of preferred and dispreferred log-probabilities is large relative to the individual gradient norms. We theoretically investigate why such inner products can be large when aligning language models and empirically validate our findings. Empirical implications of our framework extend to explaining important differences in the training dynamics of various preference optimization algorithms, and suggesting potential algorithm designs to mitigate the under-specification issue of margin-based methods and thereby improving language model alignment. 

High-Sr/Y granitoids in continental settings are sometimes erroneously regarded as the products derived from partial melting of thickened/delaminated mafic lower curst under relatively higher pressures (1.5 GPa) in a collisional orogenic setting. In fact, multiple magmatic processes in the trans-crustal magma system, such as recycling of antecrysts, crustal assimilation, and fractional crystallization, can create or modify the primary “adakitic” signature. As a result, the generation of adakitic magmas in continental settings remains controversial from a bulk-rock perspective. Here, we address the origin of adakitic plutonic rocks through geochemical and textural characterization of rock-forming minerals in the pyroxene-bearing Zhuyuan granodiorite, West Qinling, China. The Zhuyuan granodiorite formed in a post-collisional setting and primarily consists of resorbed orthopyroxene, three types of clinopyroxene, amphibole, two types of plagioclases, K-feldspar, biotite, and quartz. Type-1 Cpx has high XMg (70.0–81.7). Type-2 Cpx displays normal zoning and decreasing XMg (80.9 to 71.5) from the core to rim. Type-3 Cpx is reversely zoned, where the rims have higher XMg (75.5–86.9), Ni, Cr, suggesting a recharge event. Orthopyroxene has high-Ni and -Cr contents, as well as high XMg (80.9–82.8), indicative of antecrysts that grew in mafic magma reservoirs. The injection of magmas from different sources is supported by sieve-textured plagioclase and crystal size distributions of non-poikilitic amphibole. Finally, non-sieve textured plagioclase, biotite, K-feldspar, and quartz are late-crystallized phases, indicative of an orthocrystic origin. The melts in equilibrium with these orthocrysts display significantly higher Sr/Y values than the magma batches that crystallized other mafic phases (i.e., amphibole, clinopyroxene, and orthopyroxene). Thus, we propose that the system involved an initial high-Sr/Y melts in equilibrium with the orthocryst assemblage was generated by water-fluxed melting of intermediate to felsic sources. The addition of low Sr/Y non-orthocrysts (e.g., amphibole and pyroxene) and associated melt diluted the original “adakitic signal” in the magma reservoir and drove the bulk composition to more mafic values. Consequently, the Zhuyuan pyroxene-bearing granodiorite represents a mixture of crystals with diverse origins and distinct magma batches of various compositions (from felsic to mafic compositions). Our study emphasizes that the origin of adakitic granitoids cannot be clearly deciphered without geochemical analysis of the constituent minerals. We also suggest that Sr/Y values in plutons should be cautiously used in paleo-crustal thickness estimates in collisional settings because of possible open system scenarios as described here. 

Abstract Chromoplasts are plant organelles with a unique ability to sequester and store massive carotenoids. Chromoplasts have been hypothesized to enable high levels of carotenoid accumulation due to enhanced sequestration ability or sequestration substructure formation. However, the regulators that control the substructure component accumulation and substructure formation in chromoplasts remain unknown. In melon (Cucumis melo) fruit, β-carotene accumulation in chromoplasts is governed by ORANGE (OR), a key regulator for carotenoid accumulation in chromoplasts. By using comparative proteomic analysis of a high β-carotene melon variety and its isogenic line low-β mutant that is defective in CmOr with impaired chromoplast formation, we identified carotenoid sequestration protein FIBRILLIN1 (CmFBN1) as differentially expressed. CmFBN1 expresses highly in melon fruit tissue. Overexpression of CmFBN1 in transgenic Arabidopsis (Arabidopsis thaliana) containing ORHis that genetically mimics CmOr significantly enhances carotenoid accumulation, demonstrating its involvement in CmOR-induced carotenoid accumulation. Both in vitro and in vivo evidence showed that CmOR physically interacts with CmFBN1. Such an interaction occurs in plastoglobules and results in promoting CmFBN1 accumulation. CmOR greatly stabilizes CmFBN1, which stimulates plastoglobule proliferation and subsequently carotenoid accumulation in chromoplasts. Our findings show that CmOR directly regulates CmFBN1 protein levels and suggest a fundamental role of CmFBN1 in facilitating plastoglobule proliferation for carotenoid sequestration. This study also reveals an important genetic tool to further enhance OR-induced carotenoid accumulation in chromoplasts in crops. 

Abstract Highly-directional image artifacts such as ion mill curtaining, mechanical scratches, or image striping from beam instability degrade the interpretability of micrographs. These unwanted, aperiodic features extend the image along a primary direction and occupy a small wedge of information in Fourier space. Deleting this wedge of data replaces stripes, scratches, or curtaining, with more complex streaking and blurring artifacts—known within the tomography community as “missing wedge” artifacts. Here, we overcome this problem by recovering the missing region using total variation minimization, which leverages image sparsity-based reconstruction techniques—colloquially referred to as compressed sensing (CS)—to reliably restore images corrupted by stripe-like features. Our approach removes beam instability, ion mill curtaining, mechanical scratches, or any stripe features and remains robust at low signal-to-noise. The success of this approach is achieved by exploiting CS's inability to recover directional structures that are highly localized and missing in Fourier Space.